The tiny lesion area and the large difference between samples of neonatal punctate white matter lesion make it difficult to detect and segment the lesion. To solve the problem, a refined deep residual U-Net was proposed to realize the fine semantic segment of the lesion. Firstly, a Magnetic Resonance Imaging (MRI) image was cut into small patches. Secondly, the deep features of multiple layers of each image patch were extracted by the residual U-Net. Then, the features were fused and the probability map of the lesion distribution of each image patch was obtained. Finally, the probability map after splicing was optimized by the fully-connected condition random field to obtain the final segmentation results. The performance of the algorithm was evaluated on a dataset provided by a cooperative hospital. The results show that with only T1 order unimodal data used, the proposed model has the lesion's edge segmented more precisely, and the anti-interference ability of the model is prominent. The model has the Dice similarity coefficient of 62.51%, the sensitivity of 69.76%, the specificity of 99.96%, and the modified Hausdorff distance reduced to 33.67.